Skip to content

4.2. Testing

What should an agent test offline?

Test every deterministic boundary without paying for or depending on a model:

  • Configuration, provider selection, and direct-Ollama/gateway endpoint construction.
  • Domain parsing, ids, slugs, limits, and error contracts.
  • Seed copying, queries, transactional writes, and append-only audit triggers.
  • Tool schemas/results and conditional direct/MCP composition.
  • Skill allowlists, retrieval ranking, and workflow/delegation topology.
  • PII callbacks and safe model/tool error callbacks.
  • MCP transports and A2A card/session/task/lifespan construction.
  • Telemetry privacy defaults and call-budget policy.

Use live model evaluations for trajectory and answer behavior, not unit-test assertions over generative prose.

How does every test get isolated state?

An autouse fixture copies the immutable dataset into a temporary directory and redirects both settings:

@pytest.fixture(autouse=True)
def isolated_data_dir(tmp_path, monkeypatch):
    destination = tmp_path / "data"
    shutil.copytree(config._DEFAULT_DATA_DIR, destination)
    monkeypatch.setattr(config.settings, "data_dir", destination)
    monkeypatch.setattr(config.settings, "state_dir", tmp_path / "state")
    return destination

Tests may approve mock actions without dirtying agents/data/incidents.db or leaking state into another test.

What does a transaction test prove?

The write and audit insert must commit or roll back together. The suite creates a trigger that rejects audit insertion, calls restart_service, and verifies the service remains down. That tests the failure path a happy-path action assertion would miss.

with pytest.raises(data.DataAccessError):
    actions.restart_service("inventory")
assert data.get_service("inventory").status == "down"

When are property-based tests worth it for agents?

At the boundaries where model-generated strings enter deterministic code. A tool argument produced by a model is fuzzed input by nature — no hand-picked example list covers the unicode, control-character, and pathological-length space it can emit. tests/test_properties.py uses Hypothesis to state contracts over that whole space instead:

  • Normalizers have no third state: the output matches the strict pattern or the input is rejected.
  • Normalizers are idempotent: normalizing an accepted value again changes nothing.
  • No traversal or SQL metacharacters survive into an accepted identifier.
  • PII redaction removes deterministically-recognized entities and is stable across calls.
  • Injection neutralization never raises and marks every hit.

Randomized tests would violate this course's determinism rule, so the suite pins a fixed profile — the gate explores the space but never flakes:

hypothesis_settings.register_profile("course", derandomize=True, database=None, max_examples=200, deadline=None)
hypothesis_settings.load_profile("course")

Reserve properties for security-critical pure functions with a statable invariant. Do not use them on generative model output (no invariant to state) or where an example-based test already expresses the whole contract.

What does branch coverage enforce?

[tool.pytest.ini_options]
addopts = [
  "--cov=agent",
  "--cov-branch",
  "--cov-fail-under=95",
]

Coverage is a missing-test detector, not evidence of good assertions. The 95% branch threshold ensures error branches remain visible; reviews still judge whether the assertions express meaningful contracts.

How do you run focused and complete tests?

cd agents/python
uv run pytest tests/test_actions.py -q
mise run test

Run the smallest relevant file while developing, then the complete suite before leaving the chapter. Do not use --no-cov except the dedicated redteam task, which selects cases already covered by the full suite.

What is the testing checkpoint?

mise run test
test -z "$(git status --porcelain agents/data/incidents.db)"

Expected result: all tests pass, branch coverage is at least 95%, and the committed seed has no change.